NAS backup: support incremental backups of RBD volumes - #14386

Description

Incremental NAS backups so far relied on QEMU dirty bitmaps / libvirt checkpoints, which only
exist on file-based qcow2 storage. RBD images are raw and have nowhere to persist a bitmap, so
any Instance with a volume on Ceph was forced onto the full-only path and re-uploaded the whole
image on every cycle.

Ceph provides the equivalent natively, and this PR adds a second chain format that uses it: an
rbd snap per cycle as the anchor, plus rbd export-diff --from-snap, which computes the
changed extents inside Ceph from the image object-map.

NASBackupProvider now picks the mechanism per Instance in getIncrementalFlavor():

A mix falls back to full-only because the two chain formats cannot be combined inside one backup.
Everything else — chain identity, the nas.backup.full.every cadence, delete/tombstone semantics
and resource accounting — is shared unchanged.

Restore differs from the qcow2 path. An export-diff stream carries no pointer to the diff
before it, unlike a rebased qcow2, so the provider sends the whole chain from its full anchor up
to the backup being restored and the agent replays each diff with rbd import-diff in order. The
image is recreated first: a full diff only carries the extents that were allocated when it was
taken, so any block still allocated in the target would otherwise show through as stale data.
That also drops RBD snapshots the volume carried, which is inherent to rewinding a volume
wholesale. The restore fails closed: if the snapshots of an RBD image can not be listed it
stops, so it never removes snapshots it could not see. An image which does not exist (exit code 2)
has none to see.

Two behaviours follow from the anchor living in Ceph rather than in the QEMU process:

• a stopped Instance keeps its incremental cadence, since no live QEMU is needed — the opposite
  of the qcow2 path, where a stop forces the next backup to be a FULL;
• a chain survives an Instance restart or a host migration.

Only the newest anchor is kept per image, and a FULL additionally sweeps every older snapshot
carrying the csbackup- prefix, so a deleted chain, an unassigned offering or a restore cannot
leak snapshots into the cluster.

When the offering is removed, or the last backup of a chain is deleted, the anchors are removed
from a host which has the rbd command, and none is sent when no host has it.

Releasing the guest. While the snapshots of an Instance are taken, its filesystems are frozen
or, if there are several disks and no freeze, the Instance is paused. nasbackup.sh bounds the
freeze, pause and snapshot steps with timeouts and records what it holds in
/run/csbackup/<instance>.guest. The guest is released by

• a TERM/HUP/INT trap of the script, for a script the agent signals;
• the take-backup wrapper, when the script failed or was killed because the backup timed out;
• the agent, for every state file left when it starts, which covers SIGKILL (no trap runs), a
  crash and a restart of the agent.

A guest paused by an operator is not resumed, as the state file says what the script itself held.

One RBD backup per Instance at a time. nasbackup.sh holds a per-Instance flock
(/run/csbackup/<instance>.lock) during the RBD backup, so two backups of an Instance, such as a
scheduled one started by a management server which just restarted while the old one still runs on
the host, can not freeze the guest and share the state file at once. The refused backup exits with
code 21 and the wrapper leaves the guest to the backup which holds the lock.

The Ceph monitor list of an RBD disk is built with RbdUtils, so a bare IPv6 address is bracketed,
carries its port and survives the parsing of nasbackup.sh.

The feature stays behind the existing nas.backup.incremental.enabled zone setting, which still
defaults to false, so nothing changes for existing deployments until it is switched on.

Note: the agent drives the rbd CLI, so KVM hosts backing up Ceph volumes need ceph-common
installed.

Related: #14667, which encrypts a volume created on a storage pool with an encrypted disk
offering (4.22 base).

Test changes included here

• test_backup_recovery_nas.py: the blanket RBD skip is replaced by an _is_rbd_storage()
  predicate, so the chain tests now run on Ceph. test_stopped_vm_falls_back_to_full keeps a
  targeted skip because RBD has the opposite behaviour by design, and a new
  test_stopped_rbd_vm_keeps_incremental_cadence asserts that other side.
• The suite moves to a cloud-init template, whose kernel enumerates hot-attached virtio disks and
  which ships the usual filesystem tools, removing guest-side workarounds the legacy template
  needed.
• A new test_full_chain_restore_and_create_vm_survive_mid_chain_delete walks a whole chain:
  FULL + 3 incrementals, restore from every backup and create an Instance from every backup, then
  delete the middle incremental and repeat both sweeps over the survivors. It also asserts that an
  incremental is a small fraction of its FULL, which is what distinguishes a real export-diff
  delta from the whole image being re-exported.
• Instances created from a backup are verified by their guest contents over SSH, not only by
  reaching Running.
• Marvin gives every Instance it creates its own public IP for SSH, and destroying the Instance
  does not release it, so a class which builds many Instances from backups ran out of the public
  range of a small zone. release_publicIP() is added to marvin.lib.common and the tests release
  the IP of every Instance they create when they are done with it. The Instance shared by the class
  keeps its IP until the class ends.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested on two KVM + Ceph/RBD lab environments (zone-wide RBD primary storage with two
Ceph pools, NAS backup repository on an NFS export, nas.backup.incremental.enabled=true).

Unit tests

• NASBackupProviderTest — 60/60 pass
• LibvirtRestoreBackupCommandWrapperTest — 30/30 pass
• LibvirtTakeBackupCommandWrapperTest — 23/23 pass
• RbdBackupGuestManagerTest — 3/3 pass
• KVMPhysicalDiskTest — 5/5 pass

Smoke tests — test/integration/smoke/test_backup_recovery_nas.py

The whole suite was run repeatedly on Ceph/RBD to check for flakiness, each run against a
freshly cleaned environment:

No failures in any of these runs; failed_plus_exceptions.txt was empty every time. Runs 7 and 8
were made with the same changes on a build based on 4.23.0, after the guest release and the lock
were added; the unit tests above were run on this branch. In between, a run on a zone with only 13
free public IPs failed in the two tests which build an Instance from a backup, because every such
Instance held a public IP until the class ended; that is what the IP release above fixes. Runs 2–4 produced
identical per-test results, and runs 5–6 produced identical guest contents across all 14
verification points, so the new assertions are stable rather than lucky.

All of these pass on Ceph/RBD, where the chain tests previously could not run at all:

• test_vm_backup_lifecycle
• test_vm_backup_create_vm_from_backup
• test_incremental_chain_cadence
• test_incremental_after_vm_restart
• test_restore_from_incremental
• test_delete_middle_incremental_repairs_chain
• test_delete_full_with_children_is_deferred
• test_restore_volume_and_attach_to_vm
• test_stopped_rbd_vm_keeps_incremental_cadence
• test_full_chain_restore_and_create_vm_survive_mid_chain_delete

The two skips are deliberate, not gaps:

• test_stopped_vm_falls_back_to_full — skipped on RBD, which has the opposite correct
  behaviour. A qcow2 chain is anchored on a QEMU dirty bitmap that dies with the process, so a
  stopped instance must fall back to a FULL; an RBD chain is anchored on an rbd snap inside
  Ceph, so it keeps its cadence. The RBD side is asserted by
  test_stopped_rbd_vm_keeps_incremental_cadence, which takes an INCREMENTAL while the
  instance is stopped and then restores data written before the stop.
• test_vm_backup_create_vm_from_backup_in_another_zone — single-zone environment.

On-NAS layout and backup sizes

A FULL + 3 INCREMENTALs of an 8 GiB RBD root volume, taken manually through the API:

Each backup directory holds the diff plus the usual metadata:

<repository>/i-2-598-VM/2026.10.11.14.20.44/
    domain-config.xml                                        6989
    domblklist.xml                                            204
    domiflist.xml                                             178
    dominfo.xml                                               433
    root.ae7052e5-3244-48e9-a61c-bc6e0a14ebb0.rbddiff  1868960404

The size CloudStack records is the whole directory: .rbddiff + 7,804 bytes of XML metadata,
matching to the byte in every row above. Incrementals are 0.10 %–0.24 % of the FULL, i.e.
rbd export-diff really is shipping only the changed extents rather than re-exporting the image.

The same ratio holds over a longer run on the second environment — 24 backups taken by the
smoke suite reported FULLs of 2.18–2.25 GB against incrementals of 12.55 KB–20.82 MB, with no
backup reporting a size of 0, and every BackupAnswer carrying bitmapCreated=csbackup-<ts>
and incrementalFallback=false (no silent fallback to a full).

Restore and create-instance-from-backup, for every backup in a chain

With the FULL + 3 INCREMENTAL chain above:

• in-place restore from each of the 4 backups — all succeeded;
• create-instance-from-backup from each of the 4 backups — all 4 instances reached Running.

The automated chain test does the same thing and additionally verifies the guest: a marker file
is written before each backup, and after every restore the markers taken up to that backup must
be present and markers written after it must be absent — so a restore that silently did
nothing, or that came up with the source instance's current disk, fails instead of passing. The
instances created from backups are checked the same way over SSH, not merely for reaching
Running. Observed guest contents (7 restores and 7 created instances, each on its own IP):

restores (source instance)   [m0] → [m0,m1] → [m0,m1,m2] → [m0,m1,m2,m3]
created instances            [m0]   [m0,m1]   [m0,m1,m2]   [m0,m1,m2,m3]

Deleting a backup from the middle of the chain

An rbd export-diff stream carries no pointer to the diff before it, so a restore replays the
whole sequence from the full anchor and a diff whose descendant still needs it cannot simply be
dropped. Deleting INCREMENTAL 2 from the chain above:

• it disappeared from listBackups, while its 1.8 MB diff stayed on the repository because
  INCREMENTAL 3 still needs it, and no other backup's reported size changed;
• restoring from each of the 3 survivors still succeeded, including INCREMENTAL 3 across the
  deleted parent, which came back with all four markers — m2 included, the marker whose
  backup had been deleted;
• creating an instance from each of the 3 survivors also succeeded, and the instance built from
  INCREMENTAL 3 likewise came up carrying m2, so a freshly built volume replays across the gap
  too, not only an in-place restore.

Chain teardown leaves nothing behind

Deleting the chain afterwards swept the repository completely:

delete INCREMENTAL 3  → 2 directories removed (its own + the tombstoned INCREMENTAL 2)
delete INCREMENTAL 1  → 1 directory remaining
delete FULL           → 0 directories remaining

Deleting INCREMENTAL 3 removed two directories at once: its own, plus the INCREMENTAL 2 that had
been retained only for its sake, swept the moment it became unreferenced. After the manual chain
and six full smoke runs the instance's repository directory was empty and both Ceph pools held
0 csbackup-* snapshots across all images, so no chain leaked a snapshot into the cluster.

How did you try to break this feature and the system with this change?

• Deleted a backup from the middle of a live chain and then restored and built instances from
  every survivor, including the tip whose parent had been deleted — the case that breaks if the
  provider ever sends a chain with a hole in it.
• Deleted a FULL that still had children — it is tombstoned and hidden from the list while a
  descendant survives, and physically swept once the last descendant is deleted
  (test_delete_full_with_children_is_deferred).
• Restored the same instance repeatedly, from older and newer backups in turn, checking after
  each restore that markers written after that backup were rolled back rather than left behind.
• Stopped the instance and kept backing up — on RBD the cadence must hold instead of falling
  back to a FULL, and the resulting incremental must still restore data written before the stop.
• Restarted the instance mid-chain — an incremental taken after the restart must still be an
  INCREMENTAL and must restore correctly (test_incremental_after_vm_restart, PR feat(backup): incremental NAS backup support for KVM #13074
  regression).
• Checked the reported sizes against the repository, since an incremental that silently
  re-exported the whole image would still "work". The byte-for-byte agreement above, and the
  assertion that an incremental is a small fraction of its FULL, are what make that visible.
• Checked for leaks after every run — repository directories and csbackup-* snapshots in
  both Ceph pools.
• Released a held guest after the backup script was signalled or killed. The trap on
  TERM/HUP/INT and the per-Instance lock were exercised with a stand-in virsh; the wrapper (a
  timed-out script, a guest paused by an operator, a refused second backup) and the agent start-up
  sweep are covered by unit tests, and the sweep was also run once on a KVM host. A freeze or
  pause interrupted on a real guest by a restart of the agent in the middle of a backup has not
  been tested end to end.
• Mixed-storage instances (volumes on both RBD and file-based storage) are rejected by
  getIncrementalFlavor() and stay on the legacy full-only path, since the two chain formats
  cannot be mixed within one backup.